We consider the problem of how many samples from a Gaussian multi-index model are required to weakly reconstruct the relevant index subspace. Despite its increasing popularity as a testbed for investigating the computational complexity of neural networks, results beyond the single-index setting remain elusive. In this work, we introduce spectral algorithms based on the linearization of a message passing scheme tailored to this problem. Our main contribution is to show that the proposed methods achieve the optimal reconstruction threshold. Leveraging a high-dimensional characterization of the algorithms, we show that above the critical threshold the leading eigenvector correlates with the relevant index subspace, a phenomenon reminiscent of the Baik-Ben Arous-Peche (BBP) transition in spiked models arising in random matrix theory.

The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization. This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBMs on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning. We show in particular that RBMs reach the optimal computational weak recovery threshold, aligning with the Baik-Ben Arous-Péché (BBP) transition, in the spiked covariance model.

This paper builds a hierarchy of explicit, non-asymptotic tail bounds for the supremum of the Kostlan--Shub--Smale (KSS) random field on the sphere, and applies it to two problems: Spiked Tensor PCA and the landscape of the spherical $k$-spin model. For Tensor PCA, we study the non-asymptotic statistical limits of estimating a rank-$R$ symmetric signal tensor of order~$k\ge 3$ and dimension~$d\ge 3$ from a single Gaussian observation at signal-to-noise ratio~$λ$, through the \emph{profile maximum likelihood estimator}, the MLE restricted to normalized rank-$R$ tensors of coherence at least~$κ$. Our analysis uses a single reduction: a deterministic geometric inequality (the Tube Method) and a rank-reduction step bound the estimation error by the supremum of the canonical KSS field, which the Kac--Rice formula turns into a Gaussian integral against the expected absolute characteristic polynomial of a shifted Gaussian Orthogonal Ensemble, controlled in turn by the four explicit tail bounds of our hierarchy (three from a Mehta--Fyodorov representation, one from a Ben Arous--Dembo--Guionnet large deviation). The same reduction yields two results, each with explicit constants. For estimation, a finite-$(k,d)$ error bound recovers the asymptotically optimal rate~$\sqrt{d\log k}$ of Perry, Wein and Bandeira, with explicit dependence on the rank~$R$ and the coherence~$κ$. For the landscape, a two-sided non-asymptotic bracketing of the annealed complexity of the spherical $k$-spin Hamiltonian recovers the Auffinger--Ben Arous--Černý complexity function in the high-dimensional limit.

We study the dynamics of stochastic gradient descent (SGD) for a class of sequence models termed Sequence Single-Index (SSI) models, where the target depends on a single direction in input space applied to a sequence of tokens. This setting generalizes classical single-index models to the sequential domain, encompassing simplified one-layer attention architectures. We derive a closed-form expression for the population loss in terms of a pair of sufficient statistics capturing semantic and positional alignment, and characterize the induced high-dimensional SGD dynamics for these coordinates. Our analysis reveals two distinct training phases: escape from uninformative initialization and alignment with the target subspace, and demonstrates how the sequence length and positional encoding influence convergence speed and learning trajectories. These results provide a rigorous and interpretable foundation for understanding how sequential structure in data can be beneficial for learning with attention-based models. Stochastic Gradient Descent (SGD) is the core optimization tool driving modern machine learning. Recent years have seen substantial progress in understanding its dynamics, particularly in two-layer networks [Saad and Solla, 1995, Mei et al., 2018, Chizat and Bach, 2018, Rotskoff and VandenEijnden, 2022, Sirignano and Spiliopoulos, 2020, Arnaboldi et al., 2023a]. While global convergence is qualitatively well-understood when the network is wide enough, quantitative results are scarcer. A particularly fruitful body of recent theoretical work addressing this gap has focused on deriving precise convergence rates for particular model classes on synthetic data, such as high-dimensional Gaussian single and multi-index models [Ben Arous et al., 2021, Abbe et al., 2022, 2023].

We consider the problem of how many samples from a Gaussian multi-index model are required to weakly reconstruct the relevant index subspace. Despite its increasing popularity as a testbed for investigating the computational complexity of neural networks, results beyond the single-index setting remain elusive. In this work, we introduce spectral algorithms based on the linearization of a message passing scheme tailored to this problem. Our main contribution is to show that the proposed methods achieve the optimal reconstruction threshold. Leveraging a high-dimensional characterization of the algorithms, we show that above the critical threshold the leading eigenvector correlates with the relevant index subspace, a phenomenon reminiscent of the Baik-Ben Arous-Peche (BBP) transition in spiked models arising in random matrix theory.

The information exponent (Ben Arous et al. [2021]) and its extensions --- which are equivalent to the lowest degree in the Hermite expansion of the link function (after a potential label transform) for Gaussian single-index models --- have played an important role in predicting the sample complexity of online stochastic gradient descent (SGD) in various learning tasks. In this work, we demonstrate that, for multi-index models, focusing solely on the lowest degree can miss key structural details of the model and result in suboptimal rates. Specifically, we consider the task of learning target functions of form $f_*(x) = \sum_{k=1}^{P} \phi(v_k^* \cdot x)$, where $P \le d$, the ground-truth directions $\\{ v_k^* \\}_{k=1}^P$ are orthonormal, and the information exponent of $\phi$ is $L$. Based on the theory of information exponent, when $L = 2$, only the relevant subspace (not the exact directions) can be recovered due to the rotational invariance of the second-order terms, and when $L > 2$, recovering the directions using online SGD require $\tilde{O}(P d^{L-1})$ samples. In this work, we show that by considering both second-and higher-order terms, we can first learn the relevant space using the second-order terms, and then the exact directions using the higher-order terms, and the overall sample and complexity of online SGD is $\tilde{O}( d P^{L-1})$.

The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization. This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBMs on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning. We show in particular that RBMs reach the optimal computational weak recovery threshold, aligning with the Baik-Ben Arous-Péché (BBP) transition, in the spiked covariance model.

'Record the faces': Tense moment NBA boss gives VERY honest take on Trump attending Knicks game Leaked transcript of UNAIRED 60 Minutes interview exposes REAL reason'callous' CBS star Scott Pelley'deserved to be fired' Disgraceful texts'hot' teacher sent boy, 17, who she had illegal sex with where she moaned about her HUSBAND Everyone always said I cleared my throat a lot. But then I developed shoulder pain and doctors discovered the sinister cause... the world's deadliest cancer. Don't leave it too late like I did Outrage as Netanyahu is caught SPYING on Trump's Iran negotiators... as JD Vance reveals a chilling truth about Israel White couple gave birth to'non-Caucasian' baby. Parents were told son, 7, had ADHD... not realizing he was battling terrifying disease that has now left him BLIND'Great' mom, 32, tried to gas herself and her three young kids to death after inviting them to'popcorn sleepover' in car, prosecutors allege Medical student, 24, died by suicide in his white coat a day after he was suspended for alleged'inappropriate' behavior towards female patient, lawsuit alleges, as his heartbreaking goodbye note to parents is revealed Karmelo Anthony's parents seen leaving the courtroom in tears just before son's defense team pulls shock move Grim-faced former Louisiana mayor Misty Roberts arrives in court for sentencing after being found guilty of having sex with son's teenage friend Mother died during tummy tuck and Brazilian butt lift after clinic staff failed to hold'slow' elevator for EMTs, report alleges Gaming influencer Alex Cimo dies'very suddenly' aged 32 just a month after'refusing to accept his fate' The porn-fuelled fantasy middle-class husbands are desperate to try with their wives... and it almost always ends in divorce: JANA HOCKING All the backstage gossip from Miami Swim Week: Insider exposes'catty' VIP's diva demands... STEALING... and'morbidly embarrassing' celeb moment everyone is whispering about Girl, 13, mistakenly told she was DYING after Oregon hospital staff made jaw-dropping surgical mistake, parents' $17m lawsuit alleges Mother's final words before she was shot dead'by new husband' in front of her two young children'They have a problem with my country': Africa's best referee, who was denied entry to the US and will miss the World Cup, speaks out and insists he had a valid visa Furious dad films his partner in bed with his 19-year-old son: You've seen the viral video - now all three tell the Daily Mail what REALLY happened in the scandal gripping Australia World's shark attack hotspots revealed: As a great white is spotted in the Mediterranean, experts reveal the areas where you're most likely to be bitten The world's shark attack hotspots have been revealed, after a g reat white shark was spotted in the Mediterranean Sea. The enormous predator was recorded between Sicily and Tunisia, in what is believed to be the first ever footage captured of an adult great white in the area.

'Record the faces': Tense moment NBA boss gives VERY honest take on Trump attending Knicks game Leaked transcript of UNAIRED 60 Minutes interview exposes REAL reason'callous' CBS star Scott Pelley'deserved to be fired' Disgraceful texts'hot' teacher sent boy, 17, who she had illegal sex with where she moaned about her HUSBAND Everyone always said I cleared my throat a lot. But then I developed shoulder pain and doctors discovered the sinister cause... the world's deadliest cancer. Don't leave it too late like I did Outrage as Netanyahu is caught SPYING on Trump's Iran negotiators... as JD Vance reveals a chilling truth about Israel White couple gave birth to'non-Caucasian' baby. Parents were told son, 7, had ADHD... not realizing he was battling terrifying disease that has now left him BLIND Medical student, 24, died by suicide in his white coat a day after he was suspended for alleged'inappropriate' behavior towards female patient, lawsuit alleges, as his heartbreaking goodbye note to parents is revealed Karmelo Anthony's parents seen leaving the courtroom in tears just before son's defense team pulls shock move Grim-faced former Louisiana mayor Misty Roberts arrives in court for sentencing after being found guilty of having sex with son's teenage friend Mother died during tummy tuck and Brazilian butt lift after clinic staff failed to hold'slow' elevator for EMTs, report alleges Gaming influencer Alex Cimo dies'very suddenly' aged 32 just a month after'refusing to accept his fate' 'Great' mom, 32, tried to gas herself and her three young kids to death after inviting them to'popcorn sleepover' in car, prosecutors allege The porn-fuelled fantasy middle-class husbands are desperate to try with their wives... and it almost always ends in divorce: JANA HOCKING Meghan Markle's As Ever website has had'less than 400,000 US visitors' since January - as Duchess launches collaboration with a lifestyle influencer to plug her products Nashville's most-hated influencer sparked outrage with sick posts about teen girl who vanished into the woods after a party... now his incredible life of luxury is unraveling Girl, 13, mistakenly told she was DYING after Oregon hospital staff made jaw-dropping surgical mistake, parents' $17m lawsuit alleges Mother's final words before she was shot dead'by new husband' in front of her two young children'They have a problem with my country': Africa's best referee, who was denied entry to the US and will miss the World Cup, speaks out and insists he had a valid visa Massive twist in JPMorgan'sex slave' case as accuser unveils NEW dossier of wild claims: 'The story is about to change dramatically' A great white shark has been spotted underwater in the Mediterranean for the first time ever. Divers from Healthy Seas were removing ghost nets on an offshore shipwreck between Sicily and Tunisia when they spotted the predator.

In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.